Methods for using extracted features to perform an action

ABSTRACT

A method that extracts features from frames of a video signal to perform an action includes receiving a query related to one or more frames of a video signal; extracting one or more features of the one or more frames; identifying one or more annotations associated with the one or more frames; identifying one or more images related to the one or more frames by comparing the extracted features with reference features from a first plurality of reference works, the first plurality of reference works organized using a clustering technique based on annotations; providing the identified one or more images along with action information associated with the identified one or more images, wherein the action information is related to an action to be performed; receiving a selection of one of the identified one or more images and providing machine readable instructions to perform the action.

§ 0. RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/689,528 (incorporated herein by reference, will be issued asU.S. Pat. No. 10,063,936 on Aug. 28, 2018), titled “METHODS FOR USINGEXTRACTED FEATURE VECTORS TO PERFORM AN ACTION ASSOCIATED WITH A WORKIDENTIFIER,” filed on Aug. 29, 2017, and listing Ingemar J. Cox as theinventor, which is a continuation of U.S. patent application Ser. No.15/355,464 (incorporated herein by reference, issued as U.S. Pat. No.9,883,253 on Jan. 30, 2018), titled “METHODS FOR USING EXTRACTED FEATUREVECTORS TO PERFORM AN ACTION ASSOCIATED WITH A PRODUCT,” filed on Nov.18, 2016, and listing Ingemar J. Cox as the inventor, which is acontinuation of U.S. patent application Ser. No. 14/981,219(incorporated herein by reference, issued as U.S. Pat. No. 9,538,216 onJan. 3, 2017), titled “SYSTEM FOR TAKING ACTION WITH RESPECT TO A MEDIAWORK,” filed on Dec. 28, 2015, and listing Ingemar J. Cox as theinventor, which is a continuation of U.S. patent application Ser. No.13/830,986 (incorporated herein by reference, issued as U.S. Pat. No.9,282,359 on Mar. 8, 2016), titled “METHOD FOR TAKING ACTION WITHRESPECT TO AN ELECTRONIC MEDIA WORK,” filed on Mar. 14, 2013, andlisting Ingemar J. Cox as the inventor, which is a continuation of U.S.patent application Ser. No. 13/800,573 (incorporated herein byreference, issued as U.S. Pat. No. 8,904,464 on Dec. 2, 2014), titled“METHOD FOR TAGGING AN ELECTRONIC MEDIA WORK TO PERFORM AN ACTION,”filed on Mar. 13, 2013, and listing Ingemar J. Cox as the inventor,which is a continuation of U.S. patent application Ser. No. 13/338,079(incorporated herein by reference, issued as U.S. Pat. No. 8,640,179 onJan. 28, 2014), titled “METHOD FOR USING EXTRACTED FEATURES FROM ANELECTRONIC WORK,” filed on Dec. 27, 2011, and listing Ingemar J. Cox asthe inventor, which is a continuation of U.S. patent application Ser.No. 11/977,202 (incorporated herein by reference, issued as U.S. Pat.No. 8,205,237 on Jun. 19, 2012), titled “IDENTIFYING WORKS, USING ASUB-LINEAR TIME SEARCH, SUCH AS AN APPROXIMATE NEAREST NEIGHBOR SEARCH,FOR INITIATING A WORK-BASED ACTION, SUCH AS AN ACTION ON THE INTERNET”,filed Oct. 23, 2007, and listing Ingemar J. Cox as the inventor, whichis a continuation of U.S. patent application Ser. No. 11/445,928(incorporated herein by reference, issued as U.S. Pat. No. 8,010,988 onAug. 30, 2011), titled “USING FEATURES EXTRACTED FROM AN AUDIO AND/ORVIDEO WORK TO OBTAIN INFORMATION ABOUT THE WORK,” filed on Jun. 2, 2006,and listing Ingemar J. Cox as the inventor, which is acontinuation-in-part of U.S. patent application Ser. No. 09/950,972(incorporated herein by reference, issued as U.S. Pat. No. 7,058,223 onJun. 6, 2006), titled “IDENTIFYING WORKS FOR INITIATING A WORK-BASEDACTION, SUCH AS AN ACTION ON THE INTERNET,” filed on Sep. 13, 2001, andlisting Ingemar J. Cox as the inventor, which application claims benefitto the filing date of provisional patent application Ser. No. 60/232,618(incorporated herein by reference), titled “IDENTIFYING AND LINKINGTELEVISION, AUDIO, PRINT AND OTHER MEDIA TO THE INTERNET,” filed on Sep.14, 2000 and listing Ingemar J. Cox as the inventor.

§ 1. BACKGROUND OF THE INVENTION § 1.1 Field of the Invention

The present invention concerns linking traditional media to newinteractive media, such as that provided over the Internet for example.In particular, the present invention concerns identifying a work (e.g.,content or an advertisement delivered via print media, or via a radio ortelevision broadcast) without the need to modify the work.

§ 1.2 Related Art

§ 1.2.1 Opportunities Arising from Linking Works Delivered Via SomeTraditional Media Channel or Conduit to a More Interactive System

The rapid adoption of the Internet and associated World Wide Web hasrecently spurred interest in linking works, delivered via traditionalmedia channels or conduits, to a more interactive system, such as theInternet for example. Basically, such linking can be used to (a) promotecommerce, such as e-commerce, and/or (b) enhance interest in the workitself by facilitating audience interaction or participation. Commerceopportunities include, for example, facilitating the placement of directorders for products, providing product coupons, providing furtherinformation related to a product, product placement, etc.

In the context of e-commerce, viewers could request discount vouchers orcoupons for viewed products that are redeemable at the point ofpurchase. E-commerce applications also extend beyond advertisements. Itis now common for television shows to include product placements. Forexample, an actor might drink a Coke rather than a Pepsi brand of soda,actors and actresses might wear designer-labeled clothing such as CalvinKlein, etc. Viewers may wish to purchase similar clothing but may notnecessarily be able to identify the designer or the particular styledirectly from the show. However, with an interactive capability, viewerswould be able to discover this and other information by going to anassociated Web site. The link to this Web site can be automaticallyenabled using the invention described herein.

In the context of facilitating audience interaction or participation,there is much interest in the convergence of television and computers.Convergence encompasses a very wide range of capabilities. Although asignificant effort is being directed to video-on-demand applications, inwhich there is a unique video stream for each user of the service, aswell as to transmitting video signals over the Internet, there is alsointerest in enhancing the television viewing experience. To this end,there have been a number of experiments with interactive television inwhich viewers can participate in a live broadcast. There are a varietyof ways in which viewers can participate. For example, during gameshows, users can answer the questions and their scores can be tabulated.In recent reality-based programming such as the ABC television gameshow, “Big Brother”, viewers can vote on contestants who must leave theshow, and be eliminated from the competition.

§ 1.2.2 Embedding Work Identifying Code or Signals within Works

Known techniques of linking works delivered via traditional mediachannels to a more interactive system typically require some type ofcode, used to identify the work, to be inserted into the work before itis delivered via such traditional media channels. Some examples of suchinserted code include (i) signals inserted into the vertical blankinginterval (“VBI”) lines of a (e.g., NTSC) television signal, (ii)watermarks embedded into images, (iii) bar codes imposed on images, and(iv) tones embedded into music.

The common technical theme of these proposed implementations is theinsertion of visible or invisible signals into the media that can bedecoded by a computer. These signals can contain a variety ofinformation. In its most direct form, the signal may directly encode theURL of the associated Web site. However, since the alphanumeric stringhas variable length and is not a particularly efficient coding, it ismore common to encode a unique ID. The computer then accesses adatabase, which is usually proprietary, and matches the ID with theassociated web address. This database can be considered a form of domainname server, similar to those already deployed for network addresses.However, in this case, the domain name server is proprietary and theaddresses are unique ID's.

There are two principal advantages to encoding a proprietary identifierinto content. First, as previously mentioned, it is a more efficient useof the available bandwidth and second, by directing all traffic to asingle Web site that contains the database, a company can maintaincontrol over the technology and gather useful statistics that may thenbe sold to advertisers and publishers.

As an example of inserting signals into the vertical blanking intervallines of a television signal, RespondTV of San Francisco, Calif. embedsidentification information into the vertical blanking interval of thetelevision signal. The VBI is part of the analog video broadcast that isnot visible to television viewers. For digital television, it may bepossible to encode the information in, for example, the motion pictureexperts group (“MPEG”) header. In the USA, the vertical blankinginterval is currently used to transmit close-captioning information aswell as other information, while in the UK, the VBI is used to transmitteletext information. Although the close captioning information isguaranteed to be transmitted into the home in America, unfortunately,other information is not. This is because ownership of the verticalblanking interval is disputed by content owners, broadcasters and localtelevision operators.

As an example of embedding watermarks into images, Digimarc of Tualatin,Oreg. embeds watermarks in print media. Invisible watermarks are newerthan VBI insertion, and have the advantage of being independent of themethod of broadcast. Thus, once the information is embedded, it shouldremain readable whether the video is transmitted in NTSC, PAL or SECAManalog formats or newer digital formats. It should be more reliable thanusing the vertical blanking interval in television applications.Unfortunately, however, watermarks still require modification of thebroadcast signal which is problematic for a number of economic,logistical, legal (permission to alter the content is needed) andquality control (the content may be degraded by the addition of awatermark) reasons.

As an example of imposing bar codes on images, print advertisers arecurrently testing a technology that allows an advertisement to be shownto a camera, scanner or bar code reader that is connected to a personalcomputer (“PC”). The captured image is then analyzed to determine anassociated Web site that the PC's browser then accesses. For example,GoCode of Draper, Utah embeds small two-dimensional bar codes for printadvertisements. The latter signal is read by inexpensive barcode readersthat can be connected to a PC. AirClic of Blue Bell, Pa. provides acombination of barcode and wireless communication to enable wirelessshopping through print media. A so-called “CueCat” reads bar codesprinted in conjunction with advertisements and articles in Forbesmagazine. Similar capabilities are being tested for television and audiomedia.

Machine-readable bar codes are one example of a visible signal. Theadvantage of this technology is that it is very mature. However, thefact that the signal is visible is often considered a disadvantage sinceit may detract from the aesthetic of the work delivered via atraditional media channel or conduit.

As an example of embedding tones into music, Digital Convergence ofDallas, Tex. proposes to embed identification codes into audible musictones broadcast with television signals.

All the foregoing techniques of inserting code into a work can becategorized as active techniques in that they must alter the existingsignal, whether it is music, print, television or other media, such thatan identification code is also present. There are several disadvantagesthat active systems share. First, there are aesthetic or fidelity issuesassociated with bar codes, audible tones and watermarks. Moreimportantly, all media must be processed, before it is delivered to theend user, to contain these active signals. Even if a system isenthusiastically adopted, the logistics involved with inserting barcodes or watermarks into, say every printed advertisement, areformidable.

Further, even if the rate of adoption is very rapid, it neverthelessremains true that during the early deployment of the system, most workswill not be tagged. Thus, consumers that are early-adopters will findthat most media is not identified. At best, this is frustrating. Atworst, the naive user may conclude that the system is not reliable ordoes not work at all. This erroneous conclusion might have a veryadverse effect on the adoption rate.

Further, not only must there be modification to the production process,but modifications must also be made to the equipment in a user's home.Again, using the example of watermarking of print media, a PC must befitted with a camera and watermark detection software must be installed.In the case of television, the detection of the identification signal islikely to occur at the set-top-box—this is the equipment provided by thelocal cable television or satellite broadcasting company. In many cases,this may require modifications to the hardware, which is likely to beprohibitively expensive. For example, the audible tone used by DigitalConvergence to recognize television content, must be fed directly into asound card in a PC. This requires a physical connection between thetelevision and the PC, which may be expensive or at least inconvenient,and a sound card may have to be purchased.

§ 1.2.3 Unmet Needs

In view of the foregoing disadvantages of inserting an identificationcode into a work, thereby altering the existing signal, there is a needfor techniques of identifying a work without the need of inserting anidentification code into a work. Such an identification code can then beused to invoke a work-related action, such as work-related commercemethods and/or to increase audience interest by facilitating audienceinteraction and/or participation.

§ 2. SUMMARY OF THE INVENTION

Some embodiments consistent with the present invention provide acomputer-implemented method, apparatus, or computer-executable programsfor linking a media work to an action. Such embodiments might (a)extract features from the media work, (b) determine an identification ofthe media work based on the features extracted using a sub-linear timesearch, such as an approximate nearest neighbor search for example, and(c) determine an action based on the identification of the media workdetermined. In some embodiments consistent with the present invention,the media work is an audio signal. The audio signal might be obtainedfrom a broadcast, or an audio file format. In other embodimentsconsistent with the present invention, the media work is a video signal.The video signal might be obtained from a broadcast, or a video fileformat.

In some of the embodiments pertaining to audio files, the audio filemight be an mp3 file or some other digital representation of an audiosignal. The information might include a song title, an album title,and/or a performer name.

In some of the embodiments pertaining to video files, the video filemight be an MPEG file or some other digital representation of a videosignal. The video file might be a video work, and the information mightinclude a title of the video work, a director of the video work, andnames of performers in the video work.

§ 3. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a process bubble diagram of operations that may be performedin accordance with one version of the present invention, in whichintra-work information is used to identify the work.

FIG. 2 is a block diagram illustrating a first embodiment of the presentinvention, in which intra-work information is used to identify the work.

FIG. 3 is a block diagram illustrating a second embodiment of thepresent invention, in which intra-work information is used to identifythe work.

FIG. 4 is a block diagram illustrating a third embodiment of the presentinvention, in which intra-work information is used to identify the work.

FIG. 5 is a process bubble diagram of operations that may be performedin accordance with another version of the present invention, in whichextra-work information is used to identify the work.

FIG. 6 is a block diagram illustrating a fourth embodiment of thepresent invention, in which extra-work information is used to identifythe work.

FIG. 7 is a block diagram illustrating a fifth embodiment of the presentinvention, in which extra-work information is used to identify the work.

FIG. 8 is a block diagram illustrating an environment in which thepresent invention may operate.

FIG. 9 is an exemplary data structure in which extra-work information isassociated with a work identifier.

FIG. 10 is an exemplary data structure including work-related actions.

§ 4. DETAILED DESCRIPTION

The present invention may involve novel methods, apparatus and datastructures for identifying works without the need of embedding signalstherein. Once identified, such information can be used to determine awork-related action. The following description is presented to enableone skilled in the art to make and use the invention, and is provided inthe context of particular embodiments and methods. Various modificationsto the disclosed embodiments and methods will be apparent to thoseskilled in the art, and the general principles set forth below may beapplied to other embodiments, methods and applications. Thus, thepresent invention is not intended to be limited to the embodiments andmethods shown and the inventors regard their invention as the followingdisclosed methods, apparatus, data structures and any other patentablesubject matter to the extent that they are patentable.

§ 4.1 Functions

The present invention functions to identify a work without the need ofinserting an identification code into a work. The present invention maydo so by (i) extracting features from the work to define a featurevector, and (ii) comparing the feature vector to feature vectorsassociated with identified works. Alternatively, or in addition, thepresent invention may do so by (i) accepting extra-work information,such as the time of a query or of a rendering of the work, thegeographic location at which the work is rendered, and the station thatthe audience member has selected, and (ii) use such extra-workinformation to lookup an identification of the work. In either case, anidentification code may be used to identify the work.

The present invention may then function to use such an identificationcode to initiate a work-related action, such as for work-relatedcommerce methods and/or to increase audience interest by facilitatingaudience interaction and/or participation.

§ 4.2 Embodiments

As just introduced in § 4.1 above, the present invention may useintra-work information and/or extra-work information to identify a work.Once identified, such identification can be used to initiate an action,such as an action related to commerce, or facilitating audienceparticipation or interaction. Exemplary embodiments of the presentinvention, in which work is recognized or identified based on intra-workinformation, are described in § 4.2.1. Then, exemplary embodiments ofthe present invention, in which work is recognized or identified basedon extra-work information, are described in § 4.2.2.

§ 4.2.1 Embodiments in which Work is Recognized Based on Intra-WorkInformation, Such as a Feature Vector

Operations related to this embodiment are described in § 4.2.1.1 below.Then, various architectures which may be used to effect such operationsare described in § 4.2.1.2.

§ 4.2.1.1 Operations and Exemplary Methods and Techniques for EffectingSuch Operations

FIG. 1 is a process bubble diagram of operations that may be performedin accordance with one version of the present invention, in whichintra-work information is used to identify the work. As shown, awork-identification information storage 110 may include a number ofitems or records 112. Each item or record 112 may associate a featurevector of a work 114 with a, preferably unique, work identifier 116. Thework-identification information storage 110 may be generated by adatabase generation operation(s) 120 which may, in turn, use a featureextraction operation(s) 122 to extract features from a work at a firsttime (WORK_(@t1)), as well as a feature-to-work identification taggingoperation(s) 124.

Further, work identifier-action information storage 130 may include anumber of items or records 132. Each item or record 132 may associate a,preferably unique, work identifier 134 with associated information 136,such as an action for example. The work identifier-action informationstorage 130 may be generated by a database generation operation(s) 138which may, for example, accept manual entries.

As can be appreciated from the foregoing, the work-information storage110 records 112 and the work identification-action 130 records 132 canbe combined into a single record. That is, there need not be twodatabases. A single database is also possible in which the workidentifier, or a feature vector extracted from the work, serves as a keyand the associated field contains work-related information, such as aURL for example.

The feature extraction operation(s) 140 can accept a work, such as thatbeing rendered by a user, at a second time (WORK_(@t2)), and extractfeatures from that work. The extracted features may be used to define aso-called feature vector.

The extracted features, e.g., as a feature vector, can be used by afeature (vector) lookup operation(s) 150 to search for a matchingfeature vector 114. If a match, or a match within a predeterminedthreshold is determined, then the associated work identifier 116 isread.

The read work identifier can then be used by a work-associatedinformation lookup operation(s) 160 to retrieve associated information,such as an action, 136 associated with the work identifier. Suchinformation 136 can then be passed to action initiation operation(s) 170which can perform some action based on the associated information 136.

§ 4.2.1.1.1 Exemplary Techniques for Feature Extraction

When the user initiates a request, the specific television or radiobroadcast or printed commercial, each of which is referred to as a work,is first passed to the feature extraction operation. The work may be animage, an audio file or some portion of an audio signal or may be one ormore frames or fields of a video signal, or a multimedia signal. Thepurpose of the feature extraction operation is to derive a compactrepresentation of the work that can subsequently be used for the purposeof recognition. In the case of images and video, this feature vectormight be a pseudo-random sample of pixels from the frame or alow-resolution copy of the frame or the average intensities of n.times.nblocks of pixels. It might also be a frequency-based decomposition ofthe signal, such as produced by the Fourier, wavelet and or discretecosine transforms. It might involve principal component analysis. Itmight also be a combination of these. For television and audio signals,recognition might also rely on a temporal sequence of feature vectors.The recognition literature contains many different representations. Forblock-based methods, blocks may be accessed at pseudo-random locationsin each frame or might have a specific structure. For audio, commonfeature vectors are based on Fourier frequency decompositions, but otherrepresentations are possible. See, e.g., R. O. Duda and P. E. Hart,Pattern Classification and Scene Analysis (Wiley-Interscience, New York,1973). See also K. Fukunaga, Introduction to Statistical PatternRecognition, 2nd Ed. (Academic Press, New York, 1990). (These referencesare incorporated herein by reference.)

As previously stated, one object of the vector extraction stage is toobtain a more concise representation of the frame. For example, eachvideo frame is initially composed of 480.times.720 pixels which isequivalent to 345,600 pixels or 691,200 bytes. In comparison, anexemplary feature vector might only consist of 1 Kbyte of data.

A second purpose of the feature extraction process is to acquire arepresentation that is robust or invariant to possible noise ordistortions that a signal might experience. For example, frames of atelevision broadcast may experience a small amount of jitter, i.e.,horizontal and or vertical translation, or may undergo lossy compressionsuch as by MPEG-2. It is advantageous that these and other processes donot adversely affect the extracted vectors. For still images there hasbeen considerable work on determining image properties that areinvariant to affine and other geometric distortions. For example, theuse of Radon and Fourier-Mellin transforms have been proposed forrobustness against rotation, scale and translation, since thesetransforms are either invariant or bare a simple relation to thegeometric distortions. See, e.g., C. Lin, M. Wu, Y. M. Lui, J. A. Bloom,M. L. Miller, I. J. Cox, “Rotation, Scale, and Translation ResilientPublic Watermarking for Images,” IEEE Transactions on Image Processing(2001). See also, U.S. Pat. Nos. 5,436,653, 5,504,518, 5,582,246,5,612,729, and 5,621,454. (Each of these references is incorporatedherein by reference.)

§ 4.2.1.1.2 Exemplary Techniques for Database Generation and Maintenance

A number of possibilities exist for generating and maintaining workidentification (WID) and identification-action translation (WIDAT)databases. However, in all cases, works of interest are processed toextract a representative feature vector and this feature vector isassigned a unique identifier. This unique identifier is then enteredinto the work identification (WID) database 110 as well as into theWIDAT database 130 together with all the necessary associated data. Thisprocess is referred to as tagging. For example, in the case of anadvertisement, the WIDAT database 130 might include the manufacturer(Ford), the product name (Taurus), a product category (automotive) andthe URL associated with the Ford Taurus car together with theinstruction to translate the query into the associated URL.

The determination of all works of interest and subsequent feature vectorextraction and tagging depends on whether content owners are activelycollaborating with the entity responsible for creating and maintainingthe database. If there is no collaboration, then the database entitymust collect all works of interest and process and tag them. While thisis a significant effort, it is not overwhelming and is certainlycommercially feasible. For example, competitive market research firmsroutinely tabulate all advertisements appearing in a very wide varietyof print media. Newspapers and magazines can be scanned in and softwarealgorithms can be applied to the images to identify likelyadvertisements. These possible advertisements can then be compared withadvertisements already in the WID database 110. If there is a match,nothing further need be done. If there is not a match, the image can besent to a human to determine if the page does indeed contain anadvertisement. If so, the operator can instruct the computer to extractthe representative feature vector and assign it a unique identifier.Then, the operator can insert this information into the contentidentification database and as well as update the corresponding WIDATdatabase 130 with all the necessary associated data. This is continuallyperformed as new magazines and papers include new advertisements tomaintain the databases. This is a cost to the database entity.Television and radio broadcasts can also be monitored and, in fact,broadcast monitoring is currently performed by companies such as NielsenMedia research and Competitive Media Reporting. Television and radiobroadcasts differ from print media in the real-time nature of thesignals and the consequent desire for real-time recognition.

In many cases, advertisers, publishers and broadcasters may wish tocollaborate with the database provider. In this case, feature extractionand annotation and/or extra-work information may be performed by theadvertiser, advertisement agency, network and/or broadcaster and thisinformation sent to the database provider to update the database.Clearly, this arrangement is preferable from the database provider'sperspective. However, it is not essential.

§ 4.2.1.1.3. Exemplary Techniques for Matching Extracted Features withDatabase Entries

The extracted feature vector is then passed to a recognition (e.g.,feature look-up) operation, during which, the vector is compared toentries of known vectors 114 in a content identification (WID) database110. It is important to realize that the matching of extracted and knownvectors is not equivalent to looking up a word in an electronicdictionary. Since the extracted vectors contain noise or distortions,binary search might not be possible. Instead, a statistical comparisonis often made between an extracted vector and each stored vector. Commonstatistical measures include linear correlation and related measuressuch as correlation coefficient, but other methods can also be usedincluding mutual information, Euclidean distance and Lp-norms. Thesemeasures provide a statistical measure of the confidence of the match. Athreshold can be established, usually based on the required falsepositive and false negative rates, such that if the correlation outputexceeds this threshold, then the extracted and known vectors are said tomatch. See, e.g., R. O. Duda and P. E. Hart, Pattern Classification andScene Analysis (Wiley-Interscience, New York, 1973). See also, U.S. Pat.No. 3,919,474 by W. D. Moon, R. J. Weiner, R. A. Hansen and R. N. Linde,entitled “Broadcast Signal Identification System”. (Each of thesereferences is incorporated herein by reference.)

If binary search was possible, then a database containing N vectorswould require at most log(N) comparisons. Unfortunately, binary searchis not possible when taking a noisy signal and trying to find the mostsimilar reference signal. This problem is one of nearest neighbor searchin a (high-dimensional) feature space. In previous work, it was notuncommon to perform a linear search of all N entries, perhaps haltingthe search when the first match is found. On average, this will requireN/2 comparisons. If N is large, this search can be computationally veryexpensive.

Other forms of matching include those based on clustering, kd-trees,vantage point trees and excluded middle vantage point forests arepossible and will be discussed in more detail later. See, e.g., P. N.Yianilos “Excluded Middle Vantage Point Forests for nearest NeighborSearch”, Presented at the Sixth DIMACS Implementation Challenge: NearNeighbor Searches workshop, (Jan. 15, 1999). See also, P. N. Yianilos,“Locally lifting the curse of Dimensionality for nearest NeighborSearch” SODA 2000: 361-370. (Each of these references is incorporatedherein by reference.) Thus, for example, a sub-linear search time can beachieved. Unlike the kd-tree method which finds the nearest neighborwith certainty, randomized constructions, like the one described in P.N. Yianilos, “Locally lifting the curse of Dimensionality for nearestNeighbor Search” SODA 2000: 361-370, that succeed with some specifiedprobability may be used. One example of a sub-linear time search is anapproximate nearest neighbor search. A nearest neighbor search alwaysfinds the closest point to the query. An approximate nearest neighborsearch does not always find the closest point to the query. For example,it might do so with some probability, or it might provide any pointwithin some small distance of the closest point.

If the extracted vector “matches” a known vector in the contentidentification database, then the work has been identified. Of course,there is the risk that the match is incorrect. This type of error isknown as a false positive. The false positive rate can be reduced to anydesired value, but at the expense of the false negative rate. A falsenegative occurs when the vector extracted from a work is not matched tothe database even though the work is present in the database. There areseveral reasons why a work's feature vector may fail to match a featurevector database entry. First, the recognition system may not be capableof 100% accuracy. Second, the extracted vector will often contain noiseas a result of the transmission process. This noise may alter the valuesof a feature vector to the extent that a match is no longer possible.

Finally, there is the case where the observed work is not present in thedatabase. In this case, the work can be sent to an operator foridentification and insertion in the database.

§ 4.2.1.1.4 Exemplary Work Based Actions

Assuming that the work is correctly identified, then the identifier canbe used to retrieve associated information from the second workidentification-action translation (WIDAT) database 130 that containsinformation 136 associated with the particular work 134. Thisinformation may simply be a corresponding URL address, in which case,the action can be considered to be a form of network addresstranslation. However, in general, any information about the work couldbe stored therein, together with possible actions to be taken such asinitiating an e-commerce transaction. After looking up the workidentifier 134 in the WIDAT database 130, an action is performed onbehalf of the user, examples of which has been previously described.

In addition to using the system to allow audience members of a work toconnect to associated sites on the Internet, a number of other uses arepossible. First, the work identification database 130 allows competitivemarket research data to be collected (e.g., the action may includelogging an event). For example, it is possible to determine how manycommercials the Coca Cola Company in the Chicago market aired in themonth of June. This information is valuable to competitors such asPepsi. Thus, any company that developed a system as described abovecould also expect to generate revenue from competitive market researchdata that it gathers.

Advertisers often wish to ensure that they receive the advertising timethat was purchased. To do so, they often hire commercial verificationservices to verify that the advertisement or commercial did indeed runat the expected time. To do so, currently deployed systems by Nielsenand CMR embedded active signals in the advertisement prior to thebroadcast. These signals are then detected by remote monitoringfacilities that then report back to a central system which commercialswere positively identified. See for example U.S. Pat. No. 5,629,739 byR. A. Dougherty entitled “Apparatus and method for injecting anancillary signal into a low energy density portion of a color televisionfrequency spectrum”, U.S. Pat. No. 4,025,851 by D. E. Haselwood and C.M. Solar entitled “Automatic monitor for programs broadcast”, U.S. Pat.No. 5,243,423 by J. P. DeJean, D. Lu and R. Weissman, entitled “Spreadspectrum digital data transmission over TV video”, and U.S. Pat. No.5,450,122 by L. D. Keene entitled “In-station television programencoding and monitoring system and method”. (Each of these patents isincorporated herein by reference.) Active systems are usually preferredfor advertisement verification because the required recognition accuracyis difficult to achieve with passive systems. The passive monitoringsystem described herein supports commercial verification.

§ 4.2.1.2 Exemplary Architectures

Three alternative architectural embodiments in which the first techniquemay be employed are now described with reference to FIGS. 2, 3, and 4.

FIG. 2 is a block diagram illustrating a first embodiment of the presentinvention, in which intra-work information is used to identify the workand in which a audience member device 210, such as a PC for example,receives and renders a work that is consumed by an audience member(user). At some point, the user may wish to perform a work-specificaction such as traversing to an associated Web site. Upon initiation ofthis request, the computer 210 performs the operations 140 a, 150 a, 160a and 170 a, such as those shown in FIG. 1. To reiterate, theseoperations include a feature extraction operation(s) 140 a, featurevector lookup or matching operation(s) 150 a in connection with items orrecords 112 a in a work-identification (WID) database 110 a. If amatching feature vector 114 a is found, the work-associated informationlookup operation(s) 160 a can use the associated work identifier 116 ato accessing a work identification-action translation (WIDAT) database130 a to retrieve associated information 136 a, possibly includingdetermining what action should be performed.

As described above, the two databases might be integrated into a singledatabase. However, conceptually, they are described here as separate.

An example illustrating operations that can occur in the firstembodiment of FIG. 1, is now described. Consider a print application, inwhich say 10,000 advertisements are to be recognized that appear innational newspapers and magazines. If 1 Kbyte is required to store eachfeature vector then approximately 10 Mbytes of storage will be requiredfor the work identification database 110 a. Such a size does notrepresent a serious problem, in either memory or disk space, to presentpersonal computers.

An important issue then becomes recognition rate. While this may beproblematic, all the images are two-dimensional—three-dimensional objectrecognition is not required. Of course, since a low cost camera capturesthe printed advertisement, there may be a number of geometricdistortions that might be introduced together with noise. Nevertheless,the application is sufficiently constrained that adequate recognitionrates should be achievable with current state-of-the-art computer visionalgorithms. See, e.g., P. N. Yianilos “Excluded Middle Vantage PointForests for nearest Neighbor Search”, Presented at the Sixth DIMACSImplementation Challenge: Near Neighbor Searches workshop, Jan. 15,1999. See also, P. N. Yianilos “Locally lifting the curse ofDimensionality for nearest Neighbor Search” SODA 2000: 361-370. (Each ofthese references is incorporated herein by reference.) Thus, forexample, a sub-linear search time can be achieved. Unlike the kd-treemethod which finds the nearest neighbor with certainty, randomizedconstructions, like the one described in P. N. Yianilos, “Locallylifting the curse of Dimensionality for nearest Neighbor Search” SODA2000: 361-370, that succeed with some specified probability may be used.One example of a sub-linear time search is an approximate nearestneighbor search. Estimates of the size of the WIDAT database 130 adepend on what associated information (recall fields 136) is stored. If,for example, only a URL address is needed, about 20 characters cantypically represent most URLs. Thus, the size of the WIDAT database 130a would be less than 1 Mbyte.

The configuration just described with reference to FIG. 2 places all ofthe processing and data on each user's local machine 210. A number ofalternative embodiments, in which some or all of the storage andprocessing requirements are performed remotely, will be describedshortly.

As new works are created and made publicly available, the databasesresiding on a user's local computer become obsolete. Just as thedatabase provider 240 must continually update the databases in order toremain current, there is also a need to update local databases ondevices at audience member premises. This update process can beperformed over the Internet 230 in a manner very similar to how softwareis currently upgraded. It is not necessary to download an entirely newdatabase although this is an option. Rather, only the changes need to betransmitted. During this update process, the user's computer 210 mightalso transmit information to a central monitoring center 240 informingit of which advertisements the computer user has queried. This type ofinformation is valuable to both advertisers and publishers. Of course,care must be taken to ensure the privacy of individual users of thesystem. However, it is not necessary to know the identity of individualusers for the system to work.

FIG. 3 is a block diagram illustrating a second embodiment of thepresent invention, in which intra-work information is used to identifythe work. Although the WIDAT database can be quite small, as illustratedin the exemplary embodiment described above with respect to FIG. 2,there is still the problem of keeping this database current. Whileperiodic updates of the local databases may be acceptable, they becomeunnecessary if the WIDAT database 130 b is at a remote location 340. Inthis arrangement, illustrated in FIG. 3, after the local computer 310identifies the work, it sends a query to the remote WIDAT database 130b. The query may contain the work identifier. The remote site 340 maythen return the associated information 136. Although the remote WIDATdatabase 130 b needs to be updated by the database provider, this can bedone very frequently without the need for communicating the updates tothe local computers 310.

The second embodiment is most similar to active systems in which anembedded signal is extracted and decoded and the identifier is used tointerrogate a central database. Consequently it has many of theadvantages of such systems, while avoiding the need to insert signalsinto all works. One such advantage, is that the database providerreceives real-time information relating to users' access patterns.

The WIDAT database 130 b might physically reside at more than onelocation. In such a case, some requests will go to one site, and otherrequests will go to another. In this way, overloading of a single siteby too many users can be avoided. Other load balancing techniques arealso applicable.

FIG. 4 is a block diagram illustrating a third embodiment of the presentinvention, in which intra-work information is used to identify the work.Recall that the WIDAT database may be small relative to that workidentification database (WID). As the size of the work recognition (WID)database increases, the foregoing embodiments may become impractical.Consider, for example, a music application in which it is desired toidentify 100,000 song titles. If it is again assumed that a 1 Kbytevector can uniquely represent each song, then on the order of 100 Mbytesis now needed. This size is comparable to large application programssuch as Microsoft's Office 2000 suite. Although this still does notrepresent an inordinate amount of disk space, if this data needs toreside in memory at all times, then very few present machines will haveadequate resources. Clearly, at some point, the proposed architecturesscales to a point where requirements become impractical. In this case, afurther modification to the architecture is possible.

Since the storage and searching of the work-identifier (WID) databaserequire the most computation and storage, it may be more economical toperform these actions remotely. Thus, for example, if a user is playingan MP3 music file and wants to go to a corresponding website, the MP3file is passed to an operation that determines one or more featurevectors. In the third embodiment, instead of performing the matchinglocally 410, the one or more vectors are transmitted to a central site440 at which is stored the WID and WIDAT databases 110 c and 130 ctogether with sufficiently powerful computers to resolve this requestand those of other computer users. This configuration is illustrated inFIG. 4. Similarly, if a user is playing an MPEG or other video file andwants to initiate a work-related action, the video file is passed to anoperation 140 c that extracts one or more feature vectors. The entirevideo file need not be processed. Rather, it may be sufficient toprocess only those frames in the temporal vicinity to the users request,i.e., to process the current frame and or some number of frames beforeand after the current frame, e.g. perhaps 100 frames in all. Theextracted feature vector or feature vectors can then be transmitted to acentral site 440 which can resolve the request.

After successfully matching the feature vector, the central site 440 canprovide the user with information directly, or can direct the user toanother Web site that contains the information the user wants. In caseswhere the recognition is ambiguous, the central site 440 might returninformation identifying one of several possible matches and allow theuser to select the intended one.

The third embodiment is particularly attractive if the cost ofextracting the feature vector is small. In this case, it becomeseconomical to have feature vector extraction 140 c in digitalset-top-boxes and in video recorders 410. The latter may be especiallyuseful for the new generation of consumer digital video recorders suchas those manufactured by TIVO and Replay TV. These devices already haveaccess to the Internet via a phone line. Thus, when someone watching arecorded movie from television reacts to an advertisement, the videorecorder would extract one or more feature vectors and transmit them toa central site 440. This site 440 would determine if a match existedbetween the query vector and the database of pre-stored vectors 110 c.If a match is found, the central server 440 would transmit theassociated information, which might include a Web site address or an 800number for more traditional ordering, back to the audience user device410. Of course, a consumer device 410 such as a digital video recordermight also store personal information of the owner to facilitate onlinee-commerce. Such a device 410 could store the owner's name, address, andcredit card information and automatically transmit them to an on-linestore to complete a purchase. Very little user interaction other than toauthorize the purchase might be needed. This type of purchasing may bevery convenient to consumers.

Another advantage of the third embodiment is that it obviates the needto update local databases while, at the same time, the centrallymaintained databases can be kept current with very frequent updating.

§ 4.2.2 Embodiments in which Work is Recognized Based on Extra-WorkInformation

Operations related to this embodiment are described in § 4.2.2.1 below.Then, various architectures which may be used to effect such operationsare described in § 4.2.2.2.

If the cost of extracting a feature vector is too large, then the costof deploying any of the embodiments described in § 4.2.1 above may beprohibitive. This is particularly likely in very cost sensitive consumerproducts, including set-top-boxes and next generation digital VCR's.Acknowledging this fact, a different technique, one that is particularlywell suited for broadcasted media such as television and radio as wellas to content published in magazines and newspapers, is now described.This technique relies on the fact that a work need not be identified bya feature vector extracted from the work (which is an example of“intra-work information”), but can also be identified by when and whereit is published or broadcast (which are examples of “extra-workinformation”)

An example serves to illustrate this point. Consider the scenario inwhich a viewer sees a television commercial and responds to it. Theembodiments described in § 4.2.1 above required the user device (e.g., acomputer or set-top-box) 210/310/410 to extract a feature vector. Suchan extracted vector was attempted to be matched to another featurevector(s), either locally, or at a remote site. In the embodiments usinga remote site, if the central site is monitoring all televisionbroadcasts, then the user's query does not need to include the featurevector. Instead, the query simply needs to identify the time, geographiclocation and the station that the viewer is watching. A central site canthen determine which advertisement was airing at that moment and, onceagain, return the associated information. The same is true for radiobroadcasts. Moreover, magazines and newspapers can also be handled inthis manner. Here the query might include the name of the magazine, themonth of publication and the page number.

§ 4.2.2.1 Operations and Exemplary Methods and Techniques for EffectingSuch Operations

FIG. 5 is a process bubble diagram of operations that may be performedin accordance with another version of the present invention, in whichextra-work information is used to identify the work. As shown, a querywork-identification (QWID) information storage 510 may include a numberof items or records 512. Each item or record 512 may associateextra-work information 514, related to the work, with a, preferablyunique, work identifier 516. The query work-identification (QWID)information storage 510 may be generated by a database generationoperation(s) 520.

Further, work identifier-action information (WIDAT) storage 530 mayinclude a number of items or records 532. Each item or record 532 mayassociate a, preferably unique, work identifier 534 with associatedinformation 536, such as an action for example. The workidentifier-action (WIDAT) information storage 530 may be generated by adatabase generation operation(s) 538 which may, for example, acceptmanual entries.

As can be appreciated from the foregoing, the query work-information(QWID) storage 510 records 512 and the work identification-action(WIDAT) storage 530 records 532 can be combined into a single record.

The extra-work information aggregation (e.g., query generation)operation(s) 540 can accept a information related to a work, such as thetime of a user request or of a rendering of the work, the geographiclocation at which the work is rendered, and the station that theaudience member has selected, and generate a query from such extra-workinformation.

The query including the extra-work information can be used by a lookupoperation(s) 550 to search for a “matching” set of information 514. If amatch, or a match within a predetermined threshold is determined, thenthe associated work identifier 516 is read.

The read work identifier can then be used by a work-associatedinformation lookup operation(s) 560 to retrieve associated information,such as an action, 536 associated with the work identifier. Suchinformation 536 can then be passed to action initiation operation(s) 570which can perform some action based on the associated information 536.

If the extra-work information of a work is known (in advance),generating the query work identifier (QWID) information 510 isstraight-forward. If this were always the case, an intra-workinformation-based recognition operation would not be needed. However,very often this is not the case. For example, local televisionbroadcasts typically have discretion to insert local advertising, aswell as national advertising. Thus, it often is not possible to know inadvance when, on what station, and where a particular advertisement willplay.

In such instances, a real-time (e.g., centralized) monitoring facility580 may be used to (i) extract feature vectors from a work, (ii)determine a work identifier 116 from the extracted features, and (iii)communicate one or more messages 590 in which extra-work information(e.g., time, channel, geographic market) 592 is associated with a workidentifier 594, to operation(s) 520 for generating query workidentification (QWID) information 510.

§ 4.2.2.1.1 Exemplary Extra-Work Information

In the context of national broadcasts, geographic information may beneeded to distinguish between, for example, the ABC television broadcastin Los Angeles and that in New York. While both locations broadcastABC's programming, this programming airs at different times on the Eastand West coasts of America. More importantly, the local networkaffiliates that air ABC's shows have discretion to sell localadvertising as well as a responsibility to broadcast the nationalcommercials that ABC sells. In short, the works broadcast by ABC in LosAngeles can be different from that in other geographic locations.Geographic information is therefore useful to distinguish between thedifferent television markets. In some circumstances, geographicinformation may not be necessary, especially in parts of the world withhighly regulated and centralized broadcasting in which there are notregional differences.

§ 4.2.2.1.2 Exemplary Techniques for Generating Databases

FIG. 5 illustrates a third database 510 referred to as the query to workidentification (QWID) database. This database 510 maps the query (e.g.,in the form of time, location and channel information) into a unique IDthat identifies the perceived work. The QWID 510 and WIDAT 530 databasesmight not be separate, but for clarity will be considered so. Afterretrieving the unique work identifier 512 from the QWID database 510,the identifier can be used to access the WIDAT database 530. This isdiscussed in more detail later.

As introduced above, although it appears that this architecture does notrequire a recognition facility, such a facility may be needed. Thefeature extraction operation(s) 140 d, as well as the workidentification operation(s) 150 d and other databases 110 d, may bemoved to one or more remote sites 580.

Although TV Guide and other companies provide detailed informationregarding what will be broadcast when, these scheduling guides do nothave any information regarding what advertisements will air when. Inmany cases, this information is unknown until a day or so before thebroadcast. Even then, the time slots that a broadcaster sells to anadvertiser only provide a time range, e.g. 12 pm to 3 pm. Thus it isunlikely that all commercials and aired programming can be determinedfrom TV schedules and other sources prior to transmission. Further,occasionally programming schedules are altered unexpectedly due to livebroadcasts that overrun their time slots. This is common in sportsevents and awards shows. Another example of interrupts to scheduledprogramming occurs when a particularly important news event occurs.

During transmission, it may therefore be necessary for a central site580 to determine what work is being broadcast and to update its and/orother's database 520 accordingly based on the work identified 594 andrelevant extra-work information 592. There are a variety of ways thatthis can be accomplished.

First, it may be economically feasible to manually monitor alltelevision stations that are of interest, and manually update thedatabase with information regarding the work being monitored. In fact,Nielsen used such procedures in the early 1960's for the company totabulate competitive market data. More than one person can be employedto watch the same channel in order to reduce the error rate. It shouldbe noted that the recent ruling by the FCC that satellite broadcasterssuch as DirecTV, DishTV and EchoStar can carry local stationssignificantly reduces the cost of monitoring many geographic markets.Currently, DirecTV, for example, carries the four main local stations ineach of the 35 largest markets. Thus, these 4.times.35=140 channels canall be monitored from a single site 580. This site would be providedwith satellite receivers to obtain the television channels.

Unfortunately, however, humans are error prone and the monitoring ofmany different stations from many different geographic locations can beexpensive. In order to automate the recognition process, a central site580 could employ a computer-based system to perform automaticrecognition. Because the recognition is centralized, only one or a fewsites are needed. This is in comparison with the first architecture wedescribed in which a complete recognition system was required in everyuser's home or premise. This centralization makes it more economic toemploy more expensive computers, perhaps even special purpose hardware,and more sophisticated software algorithms. When video frames or clipscannot be identified or are considered ambiguous, this video can bequickly passed to human viewers to identify. Further, it should bepossible for the automated recognition system to use additionalinformation such as television schedules, time of day, etc in order toimprove its recognition rate.

§ 4.2.2.1.2 Exemplary Techniques for Generating Queries Based onExtra-Work Information

At the audience member (user) premises, all that is needed is for thedevice to send a query to a database-server with information thatincludes extra-work information, such as geographic location, time andchannel. Usually, this extra-work information would be transmitted inreal-time, while the work (e.g., an advertisement) is being broadcast.However, this is not necessary. If the television does not have accessto the Internet, and most TV's do not yet, then an audience member(user) may simply remember or record which channel he or she was viewingat what time. In fact, the user device could store this information forlater retrieval by the user. At a convenient later time, the user mightaccess the Internet using a home PC. At this time, he or she can querythe database by entering this extra-work information (e.g., togetherwith geographic information) into an application program or a webbrowser plug-in.

Another possibility is allowing an audience member (user), at the timehe or she is consuming (e.g., viewing, reading, listening to, etc.) thework, to enter query information into a handheld personal digitalassistant (“PDA”) such as a Palm Pilot, so as not to forget it. Thisinformation can then be manually transferred to a device connected to anetwork, or the information can be transferred automatically using, forexample, infrared communications or via a physical link such as acradle. Recently, PDAs also have some wireless networking capabilitiesbuilt in, and thus might support direct access to the informationdesired. Further, software is available that allows a Palm Pilot orother PDA to function as a TV remote control device. As such, the PDAalready knows the time of day and channel being viewed. It also probablyknows the location of the audience member, since most PDA users includetheir own name and address in the PDA's phonebook and identify it astheir own. Thus, with one or a few clicks, an audience member PDA usercould bookmark the television content he or she is viewing. If the PDAis networked, then the PDA can, itself, retrieve the associatedinformation immediately. Otherwise, the PDA can transfer this bookmarkeddata to a networked device, which can then provide access to the centraldatabase.

§ 4.2.2.2 Exemplary Architectures

FIG. 6 is a block diagram illustrating a fourth embodiment of thepresent invention, in which extra-work information is used to identifythe work. As shown, an extra-work information aggregation operation 540a may be effected on a device 610, such as a PC, at the audience member(user) premises. The various databases 510 a, 530 a, and 110 e, as wellas the database generation operation(s) 520 a/538 a, the lookupoperation(s) 550 a and the work-associated information lookupoperation(s) 560 a may be provided at one or more centralized monitoringand query resolution centers 640.

FIG. 7 is a block diagram illustrating a fifth embodiment of the presentinvention, in which extra-work information is used to identify the work.This fifth embodiment is similar to the fourth embodiment illustrated inFIG. 6 but here, the monitoring center 740 a and query resolution center740 b are separate.

These embodiments have many advantages for television and radiobroadcasters who desire to provide Internet links or other action.First, the audience member (user) equipment, whether it is a computer,set-top-box, television, radio, remote control, personal digitalassistant (pda), cell phone or other device, does not need to performany processing of the received signal. As such, there is almost no costinvolved to equipment manufacturers.

These last embodiments have some similarity with services such as thoseprovided by the companies Real Names of Redwood City, Calif., AmericaOnline (“AOL”) and especially iTag from Xenote. The popular press hasreported on the difficulties associated with assigning domain names. Thesimplest of these problems is that almost all the one-word names in the“.com” category have been used. Consequently, domain names can often bedifficult to remember. To alleviate this problem, RealNames and AOLprovide alternative, proprietary name spaces (AOL calls these keywords).For a fee, a company may register a name with these companies. Thus,rather than type the URL http://www.bell-labs.com, the simple keyword“bell” might be sufficient to access the same Web site. Thesecapabilities are convenient to users. However, these systems are verydifferent from the fourth and fifth embodiments described. First, andforemost, these systems are not designed to identify content. Rather,they are simply alternative network address translation systems based oneasily remembered mnemonics which are sold to interested companies. Assuch, the user is still expected to type in an address, but this addressis easier to remember than the equivalent URL. In contrast, while a usermay manually enter the information describing the work, the preferredembodiment is for the computer, set-top-box or other device toautomatically generate this information. Further, the mapping ofkeywords to network addresses is an arbitrary mapping maintained by AOLor Real Names. For example, the keyword “bell” might just as reasonablypoint to the Web site for Philadelphia's Liberty Bell as to Lucent'sBell Labs. In contrast, the query used in the fourth and fifthembodiments is designed to contain all the necessary data to identifythe work, e.g. the time, place and television channel during which thework was broadcast. There is nothing arbitrary about this mapping. Itshould also be pointed out that the proposed system is dynamic—the samework, e.g. a commercial, potentially has an infinite number of addressesdepending on when and where it is broadcast. If an advertisement airs100,000 unique times, then there are 100,000 different queries thatuniquely identify it. Moreover, the exemplary query includes naturallyoccurring information such as time, place, channel or page number. Thisis not the case for AOL or RealNames, which typically assigns one ormore static keywords to the address of a Web site.

Xenote's iTag system is designed to identify radio broadcasts and uses aquery similar to that which may be used in the fourth and fifthembodiments, i.e. time and station information. However, the workidentification information is not dynamically constructed but is insteadbased on detailed program scheduling that radio stations must provideit. As such, it suffers from potential errors in scheduling and requiresthe detailed cooperation of broadcasters. While the fourth and fifthembodiments might choose to use program scheduling information and otherancillary information to aid in the recognition process, they do notexclusively rely on this. The concept of resolving a site name byrecognizing the content is absent from the above systems.

§ 4.2.3 Exemplary Apparatus for Audience Member (User) Premise Device

While personal computers may be the primary computational device at auser's location, it is not essential to use a PC. This is especiallytrue of the embodiments depicted in FIGS. 6 and 7, which do not requirethe content, e.g. video signal, to be processed. Instead, only a uniqueset of identification parameters such as time, location and channel areprovided to identify the perceived Work. Many forms of devices cantherefore take advantage of this configuration.

As previously noted, personal digital assistants (PDAs) can be used torecord the identification information. This information can then betransferred to a device with a network communication such as a PC.However, increasingly, PDAs will already have wireless networkcommunication capabilities built-in, as with the Palm VII PDA. Thesedevices will allow immediate communication with the query resolutioncenter and all information will be downloaded to them or they canparticipate in facilitating an e-commerce transaction. Similarly,wireless telephones are increasingly offering web-enabled capabilities.Consequently, wireless phones could be programmed to act as a userinterface.

New devices can also be envisaged, including a universal remote controlfor home entertainment systems with a LCD or other graphical display anda network connection. This connection may be wireless or the remotecontrol might have a phone jack that allows it to be plugged directlyinto an existing phone line. As home networks begin to be deployed, suchdevices can be expected to communicate via an inexpensive interface tothe home network and from there to access the Internet.

In many homes, it is not uncommon for a computer and television to beused simultaneously, perhaps in the same room. A person watchingtelevision could install a web browser plug-in or applet that would askthe user to identify his location and the station being watched. Then,periodically, every 20 seconds for example, the plug-in would update alist of web addresses that are relevant to the television programs beingwatched, including the commercials. The audience member would thensimply click on the web address of interest to obtain furtherinformation. This has the advantage that the viewer does not have toguess the relevant address associated with a commercial and, in fact,can be directed to a more specialized address, such aswww.fordvehicles.com/ibv/tausrus2kflash/flash.html, rather than thegeneric www.ford.com site. Of course, this applet or plug-in could alsoprovide the database entity with information regarding what is beingaccessed from where and at what time. This information, as notedearlier, is valuable to advertisers and broadcasters. For PC's that haveinfra-red communication capabilities, it is straightforward to eithercontrol the home entertainment center from the PC or for the PC todecode the signals from a conventional remote control. Thus, as a userchanges channels, the PC is able to automatically track the channelchanges.

Recording devices such as analog VCR's and newer digital recordingdevices can also be exploited in the embodiments depicted in FIGS. 6 and7, especially if device also record the channel and time information forthe recorded content. When a user initiates a query, the recorded timeand channel, rather than the current time and channel, then form part ofthe identification information.

Digital set-top-boxes are also expected to exploit the capabilitiesdescribed herein. In particular, such devices will have two-waycommunication capabilities and may even include cable modem capabilitiesof course, the two-way communication need not be over a televisioncable. For example, satellite set-top-boxes provide up-linkcommunications via a telephone connection. Clearly, such devices providea convenient location to enable the services described herein. Moreover,such services can be provided as part of the OpenCable and DOCSIS (dataover cable service interface specification) initiatives.

§ 4.2.4 Information Retrieval Using Features Extracted from Audio and/orVideo Works

Some embodiments consistent with the present invention provide acomputer-implemented method, apparatus, or computer-executable programfor providing information about an audio file or (a video file) playedon a device. Such embodiments might (a) extract features from the audio(or video) file, (b) communicate the features to a database, and (c)receive the information about the audio (or video) file from thedatabase. In some embodiments consistent with the present invention, theact of extracting the features is performed by a microprocessor of thedevice, and/or a digital signal processor of the device. The receivedinformation might be rendered on an output (e.g., a monitor, a speaker,etc.) of the device. The received information might be stored (e.g.,persistently) locally on the device. The information might be stored ona disk, or non-volatile memory.

In some of the embodiments pertaining to audio files, the audio filemight be an mp3 file or some other digital representation of an audiosignal. The information might include a song title, an album title,and/or a performer name.

In some of the embodiments pertaining to video files, the video filemight be an MPEG file or some other digital representation of a videosignal. The video file might be a video work, and the information mightinclude a title of the video work, a director of the video work, andnames of performers in the video work.

§ 4.3 Operational Examples

An example illustrating operations of an exemplary embodiment of thepresent invention, that uses intra-work information to identify thework, is provided in § 4.3.1. Then, an example illustrating operationsof an exemplary embodiment of the present invention, that usesextra-work information to identify the work, is provided in § 4.3.2.

§ 4.3.1 Operational Example where Intra-Work Information is Used toIdentify the Work

A generic system for monitoring television commercials is now described.Obviously, the basic ideas extend beyond this specific application.

The process of recognition usually begins by recognizing the start of acommercial. This can be accomplished by looking for black video framesbefore and after a commercial. If a number of black frames are detectedand subsequently a similar number are detected 30 seconds later, thenthere is a good chance that a commercial has aired and that others willfollow. It is also well known than the average sound volume duringcommercials is higher than that for television shows and this too can beused as an indicator of a commercial. Other methods can also be used.The need to recognize the beginning of a commercial is not essential.However, without this stage, all television programming must be assumedto be commercials. As such, all video frames must be analyzed. Theadvantage of determining the presence of a commercial is that less videocontent must be processed. Since the percentage of advertising time isrelatively small, this can lead to considerable savings. For example,commercials can be buffered and then subsequently processed while thetelevision show is being broadcast. This reduces the real-timerequirements of a system at the expense of buffering, which requiresmemory or disk space. Of course, for the applications envisioned herein,a real-time response to a user requires real-time processing.

Once it is determined that an advertisement is being broadcast, it isnecessary to analyze the video frames. Typically, a compactrepresentation of each frame is extracted. This vector might be apseudo-random sample of pixels from the frame or a low-resolution copyof the frame or the average intensities of n.times.n blocks of pixels.It might also be a frequency-based decomposition of the signal, such asproduced by the Fourier, Fourier-Mellin, wavelet and or discrete cosinetransforms. It might involve principal component analysis or anycombination thereof. The recognition literature contains many differentrepresentations. For block-based methods, the n.times.n blocks may belocated at pseudo-random locations in each frame or might have aspecific structure, e.g. a complete tiling of the frame. The featurevector might then be composed of the pixels in each block or someproperty of each block, e.g. the average intensity or a Fourier or otherdecomposition of the block. The object of the vector extraction stage isto obtain a more concise representation of the frame. Each frame isinitially composed of 480.times.720 pixels which is equivalent to345,600 bytes, assuming one byte per pixel. In comparison, the featurevector might only consist of 1 Kbyte of data. For example, if each frameis completely tiled with 16.times.16 blocks, then the number of blocksper frame is 345,600/256=1350. If the average intensity of each blockconstitutes the feature vector, then the feature vector consists of 1350bytes, assuming 8-bit precision for the average intensity values.Alternatively, 100 16.times.16 blocks can be pseudo-randomly located oneach frame of the video. For each of these 100 blocks, the first 10 DCTcoefficients can be determined. The feature vector then consists of the100.times.10=1000 DCT coefficients. Many other variations are alsopossible. In many media applications, the content possesses strongtemporal and spatial correlations. If necessary, these correlations canbe eliminated or substantially reduced by pre-processing the contentwith a whitening filter.

A second purpose of the feature extraction process is to acquire arepresentation that is robust or invariant to possible noise ordistortions that a signal might experience. For example, frames of atelevision broadcast may experience a small amount of jitter, i.e.horizontal and or vertical translation, or may undergo lossy compressionsuch as MPEG-2. It is advantageous, though not essential, that these andother processes do not adversely affect the extracted vectors.

Each frame's feature vector is then compared with a database of knownfeature vectors. These known vectors have previously been entered into acontent recognition database together with a unique identifier. If aframe's vector matches a known vector, then the commercial isrecognized. Of course, there is the risk that the match is incorrect.This type of error is known as a false positive. The false positive ratecan be reduced to any desired value, but at the expense of the falsenegative rate. A false negative occurs when a frame's vector is notmatched to the database even though the advertisement is present in thedatabase. There are several reasons why a frame's feature vector mayfail to match. First, the recognition system may not be capable of 100%accuracy. Second, the extracted vector will contain noise as a result ofthe transmission process. This noise may alter the values of a featurevector to the extent that a match is no longer possible. Finally, thereis the case where the observed commercial is not yet present in thedatabase. In this case, it is necessary to store the commercial and passit (e.g., to a person) for identification and subsequent entry in thedatabase.

It is important to realize that the matching of extracted and knownvectors is not equivalent to looking up a word in an electronicdictionary. Since the extracted vectors contain noise or distortions,binary search is often not possible. Instead, a statistical comparisonis often made between an extracted vector and each stored vector. Commonstatistical measures include linear correlation and related measuressuch as correlation coefficient, but other methods can also be used,including clustering techniques. See, e.g., the Duda and Hart reference.These measures provide a statistical measure of the confidence of thematch. A threshold can be established, usually based on the requiredfalse positive and negative rates, such that if the correlation outputexceeds this threshold, then the extracted and known vectors are said tomatch.

If binary search was possible, then a database containing N vectorswould require at most log(N) comparisons. However, in currentadvertisement monitoring applications there is no discussion ofefficient search methods. Thus, a linear search of all N entries may beperformed, perhaps halting the search when the first match is found. Onaverage, this will require N/2 comparisons. If N is large, this can becomputationally expensive. Consider a situation in which one out of100,000 possible commercials is to be identified. Each 30-secondcommercial consists of 900 video frames. If all 900 frames are stored inthe database, then N=90,000,000. Even if only every 10^(th) video frameis stored in the database, its size is still nine million. Whiledatabases of this size are now common, they rely of efficient search toaccess entries, i.e., they do not perform a linear search. A binarysearch of a 90,000,000-item database requires less than 20 comparisons.In contrast, a linear search will require an average of 45,000,000!

With 9 million entries, if each vector is 1 Kbyte, then the storagerequirement is 9 Gigabytes. Disk drives with this capacity are extremelycheap at this time. However, if the database must reside in memory dueto real-time requirements, then this still represents a substantialmemory requirement by today's standards. One reason that the data mayneed to be stored in memory is because of the real-time requirements ofthe database. If 10 channels are being simultaneously monitored withineach of 50 geographic areas, then there will be 15,000 queries persecond to the content recognition database, assuming each and everyframe is analyzed. This query rate is low. However, if a linear searchis performed then 675 billion comparisons per second will be required.This is an extremely high computational rate by today's standards. Evenif only key frames are analyzed, this is unlikely to reduce thecomputational rate by more than an order of magnitude.

If an advertisement is not recognized, then typically, the remotemonitoring system will compress the video and transmit it back to acentral office. Here, the clip is identified and added to the databaseand the remote recognition sites are subsequently updated.Identification and annotation may be performed manually. However,automatic annotation is also possible using optical characterrecognition software on each frame of video, speech recognitionsoftware, close captioning information and other information sources. Asthese methods improve in accuracy, it is expected that they will replacemanual identification and annotation.

The recognition system described can be considered to be a form ofnearest neighbor search in a high dimensional feature space. Thisproblem has been very well studied and is known to be very difficult asthe dimensionality of the vectors increases. A number of possible datastructures are applicable including kd-trees and vantage point trees.These data structures and associated search algorithms organize aN-point dataset (N=90,000,000 in out previous example) so thatsub-linear time searches can be performed on average. However,worst-case search times can be considerably longer. Recently, Yianilosproposed an excluded middle vantage point forest for nearest neighborsearch. See, e.g., the Yianilos reference. This data structureguarantees sub-linear worst-case search times, but where the search isnow for a nearest neighbor within a fixed radius, .tau. The fixed radiussearch means that if the database contains a vector that is within .tau.of the query, then there is a match. Otherwise, no match is found. Incontrast, traditional vantage point trees will always return a nearestneighbor, even if the distance between the neighbor and the query isvery large. In these cases, if the distance between the query and thenearest neighbor exceeds a threshold, then they are considered not tomatch. This is precisely what the excluded middle vantage point forestimplicitly does.

Using an excluded middle vantage point forest, will allow accuratereal-time recognition of 100,000 broadcasted advertisements. Thisentails constructing an excluded middle vantage point forest based onfeature vectors extracted from say 90,000,000 frames of video. Ofcourse, using some form of pre-filtering that eliminates a large numberof redundant frames or frames that are not considered to be good uniqueidentifiers can reduce this number. One such pre-filter would be to onlyexamine the I-frames used when applying MPEG compression. However, thisis unlikely to reduce the work identification database (WID) size bymore than one order of magnitude. Assuming 10 channels are monitored ineach of 50 geographic regions, then the query rate is15,000=10.times.50.times.30 queries per second.

§ 4.3.2 Operational Example where Extra-Work Information is Used toIdentify the Work

FIG. 8 depicts a satellite television broadcast system 800, though cableand traditional broadcast modes are also applicable. Block 810represents audience members (users) watching a TV channel in their home,which also has a connection 812 to the Internet 820. Other networks arealso possible. The satellite broadcasts are also being monitored by oneor more television monitoring centers 840 a. These centers 840 a maymonitor all or a subset of the television channels being broadcast. Theyare not restricted to monitoring satellite TV broadcasts but may alsomonitor cable and traditional terrestrial broadcasts. The primarypurpose of these monitoring centers 840 a is to identify the works beingbroadcasted. Of particular interest are television advertisements.However, other works, or portions thereof, may also be identified. Eachtime a new segment of a work is identified, the monitoring system orsystems 840 a update one or more database centers 840 b, informing themof the time, place, channel and identity of the identified segment. Thesegment may be a complete thirty second commercial or, more likely,updates will occur more frequently, perhaps at a rate of 1 update persecond per channel per geographic location. The database center 840 bupdates its database so that queries can be efficiently responded to insub-linear time.

The database centers 840 b can use traditional database technology. Ingeneral, the query search initiated by an audience member is not anearest neighbor search but can be a classical textual search proceduresuch as a binary search. The nearest neighbor search is appropriate forthe monitoring sub-system 840 a. The database centers 840 b arecontinually updated as each new advertisement, television show orportion thereof is recognized. Standard updating algorithms can be used.However, random new entries to the database are unlikely. Rather, eachnew entry, or set of entries, denotes a new time segment that is laterthan all previously inserted items. As such, each new entry can beappended to the end of the database while still maintaining an ordereddata structure that is amenable to binary and other efficient searchtechniques. If two entries have the same time in their time field, itemscan be sorted based on secondary fields such as the channel andgeographic location, as depicted in FIG. 9. Since the number of suchentries will be relatively small compared with the entire database, itmay be sufficient to simply create a linear linked list of such entries,as depicted in FIG. 9. Of course, the size of the database is constantlyincreasing. As such, it may become necessary to have several levels ofstorage and caching. Given the envisaged application, most user querieswill be for recent entries. Thus, the database may keep the last hoursworth of entries in memory. If there is one entry per second for each of100 channels in 100 geographic locations, this would correspond to3600.times.100.times.100=36,000,000 entries which is easily accommodatedin main memory. Entries that are older than one hour may be stored ondisk and entries older than one week may be archived (e.g., backed up ontape) for example. The entries to this database can include time,location and channel information together with a unique identifier thatis provided by the monitoring system. Of course, additional fields foreach entry are also possible.

When a user query is received, the time, channel and geographicinformation are used to retrieve the corresponding unique identifierthat is then used to access a second database that contains informationassociated with the identified work.

An entry 1000 in this second database is depicted in FIG. 10, whichshows that associated with the unique identifier 1010, the name of aproduct 1020, a product category 1030, the manufacturer 1040 and thecommercial's associated web site 1050. Many other data fields 1060 arealso possible. Such additional fields may include fields that indicatewhat action should be taken on behalf of the requesting user. Exampleactions include simply redirecting a request to an associated Web site,or initiating an e-commerce transaction or providing an associatedtelephone number that may be automatically dialed if the querying deviceis a cell phone or displaying additional information to the user. Thisdatabase is likely to be updated much less frequently, perhaps only asoften as once or twice a day, as batches of new advertisements are addedto the system. Alternatively, it might be updated as each newadvertisement is added to the system.

An audience member (user) 810 watching a television commercial forexample may react to the advertisement by initiating a query to thedatabase center 840 b. The device whereby the user initiates the querymight be a television or set-top-box remote control, or a computer or awireless PDA or a (WAP-enabled) cell phone or a specialized device.Typically, the query will occur during the airing of the commercial or ashortly thereafter. However, the time between the broadcasting of theadvertisement and the time of the associated query is not critical andcan, in some instances be much longer. For example, the audience membermight bookmark the query information in a device such as a PDA or aspecialized device similar to those developed by Xenote for their Itagradio linking. Later, the audience member may transmit the query to thedatabase center 840 b. This might happen hours or even days later.

The query contains information that the database center 840 b uses toidentify the work being viewed. This information might include the timeand place where the audience member was, together with the channel beingviewed. Other identifying information is also possible. The query mayalso contain additional information that may be used to facilitate theuser's transaction and will include the return address of the user. Forexample, if the user is intending to order a pizza after seeing a PizzaHut advertisement, the query may also contain personal informationincluding his or her identity, street address and credit cardinformation.

When the database center 840 b receives a query, data in the query isused to identify the work and associated information. A number ofpossible actions are possible at this point. First, the database center840 b may simply function as a form of proxy server, mapping theaudience member's initial query into a web address associated with theadvertisement. In this case, the audience member will be sent to thecorresponding Web site. The database center 840 b may also sendadditional data included in the initial query to this Web site 850 inorder to facilitate an e-commerce transaction between the audiencemember and the advertiser. In some cases, this transaction will not bedirect, but may be indirect via a dealer or third party applicationservice provider. Thus, for example, though an advertisement by FordMotor Company may air nationally, viewers may be directed to differentWeb sites for Ford dealerships depending on both the audience member'sand the dealerships' geographic locations. In other cases, advertisersmay have contracted with the database center 840 b to provide e-commercecapabilities. This latter arrangement has the potential to reduce theamount of traffic directed over the public Internet, restricting it,instead to a private network associated with the owner of the databasecenter.

If the audience member (user) is not watching live television but isinstead watching a taped and therefore time-shifted copy, thenadditional processes are needed. For the new generation of digital videorecorders, irrespective of the recording media (tape or disk), it islikely to be very easy to include information identifying the locationof the recorder, as well as the time and channel recorded. Locationinformation can be provided to the recorder during the setup andinstallation process, for example. Digital video recorders, such asthose currently manufactured by TIVO of Alviso, Calif. or Replay TV ofSanta Clara, Calif. have a network connection via telephone, which canthen send the query of an audience member to the database center 840 busing the recorded rather than the current information.

In cases where query information has not been recorded, it is stillpossible to initiate a successful query. However, in this case, it maybe necessary to extract the feature vector from the work of interest andsend this information to the monitoring center 840 a where the featurevector can be identified. This form of query is computationally moreexpensive but the relative number of such queries compared to those sentto the database centers 840 b is expected to be small. It should also benoted that the physical separation of the monitoring and databasecenters, depicted in FIGS. 6 and 7, is not crucial to operation of thesystem and simply serves to more clearly separate the differentfunctionality present in the overall system configuration.

Although the implementation architectures described above focus on thetelevision media, it is apparent that the present invention isapplicable to audio, print and other media.

§ 4.4 Conclusions

None of the embodiments of the invention require modification to thework or content, i.e., no active signal is embedded. Consequently, thereis no change to the production processes. More importantly, from a userperspective, deployment of this system need not suffer from poor initialcoverage. Provided the database is sufficiently comprehensive, earlyadopters will have comprehensive coverage immediately. Thus, there isless risk that the consumer will perceive that the initial performanceof the deployed system is poor. Further, the present invention permitsstatistics to be gathered that measure users' responses to content. Thisinformation is expected to be very useful to advertisers and publishersand broadcasters.

What is claimed is:
 1. A method comprising: (a) receiving, at a computersystem from a user electronic device, a query related to one or moreframes from a video signal displayed at the user electronic device; (b)extracting, using the computer system, one or more features of the oneor more frames, wherein the extracted features include a compactrepresentation of at least a portion of the one or more frames relatedto one or more blocks of pixels of the frame; (c) identifying, using thecomputer system, one or more annotations associated with the one or moreframes; (d) identifying, at the computer system, one or more imagesrelated to the one or more frames by comparing the extracted featuresfrom the one or more frames with reference features from a firstplurality of reference works using a sub-linear search of referencefeatures that identifies one or more matches to the one or more framesbut does not guarantee to identify an exact match to the one or moreframes, wherein the first plurality of reference works are organizedusing a clustering technique based on annotations including the one ormore annotations to eliminate a second plurality of reference works frombeing searched; (e) providing, by the computer system to the userelectronic device, the identified one or more images along with actioninformation associated with the identified one or more images, whereinthe action information is related to an action to be performed at theuser electronic device; (f) receiving, at the computer system from theuser electronic device, a selection of one of the identified one or moreimages; and (g) providing, to the electronic device, machine readableinstructions for performance of the action associated with the selectedone of the identified one or more images.
 2. The method of claim 1,wherein the extracted features comprise a low-resolution copy of the atleast the portion of the one or more frames.
 3. The method of claim 1,wherein the extracted features comprise average intensities of n times nblocks of pixels of the at least the portion of the one or more frames.4. The method of claim 1, wherein the extracted features comprise apseudo-random sample of pixels from the at least the portion of the oneor more frames.
 5. The method of claim 1, wherein step (c) is performedby using optical character recognition software on the one or moreframes.
 6. The method of claim 1, wherein step (c) is performed by usingclosed captioning information.
 7. The method of claim 1, wherein thefirst plurality of reference works comprise content delivered viatelevision broadcast.
 8. The method of claim 1, wherein the actionassociated with the selected one of the identified one or more images isredirecting a request to an associate website by a network addresstranslation.
 9. The method of claim 1, wherein the action associatedwith the selected one of the identified one or more images is displayingadditional information to the user.
 10. The method of claim 1, whereinthe action associated with the selected one of the identified one ormore images is automatically dialing a telephone number.
 11. The methodof claim 1, wherein the user electronic device is a personal computer.12. The method of claim 1, wherein the user electronic device is a cellphone.
 13. The method of claim 1, wherein the user electronic device isa set top box.